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After-School Math 

A math program, adapted for use in the af- 
ter-school setting, resulted in 49 more hours 
of math instruction during the school year 
(30 percent more) for math program stu- 
dents than for their counterparts. The math 
program students scored 2.8 scaled points 
higher on a test of math achievement at the 
end of one year, an 8.5 percent difference 
in achievement growth. The impacts did 
not vary significantly for students in differ- 
ent grades or with different prior levels of 
achievement. Nor did the math program 
students and their counterparts differ sig- 
nificantly on other in-school behavior. 




After-school programs receive federal support through the 
21st Century Community Learning Centers, established in 
1999. A primary purpose of the program is to provide op- 
portunities for academic enrichment to help students meet 
state and local standards in core content areas. Findings 
from a previous national evaluation of the program indicate 
that the program grants awarded between 1999 and 2002 
had, on average, a limited academic impact on the academic 
achievement of participating elementary school students. 1 

A possible factor is that most academic activities at the 
evaluation sites consisted of homework sessions in which 
students received limited additional academic assistance 
(such as instruction or assistance with homework). In ad- 
dition, attendance was limited and sporadic. But analyses 
comparing the academic outcomes of frequent and infre- 
quent participants suggest that increasing attendance alone 
is unlikely to improve the academic findings. So, the lim- 
ited academic effects and the low levels of formal academic 
assistance offered in these programs highlight the need for 
better academic programming. In response, the Institute of 
Education Sciences supported the development and evalu- 
ation of instructional resources in core content areas that 
could be used in after-school programs. 



This brief presents findings for the first of two years of pro- 
gram operations (school year 2005-06) on a study for math. 

The math program 

The curriculum developer — Harcourt School Publishers — 
was selected through a competitive process to adapt its 
school-day materials for use in the after-school setting. 

The developer was asked to create material that is engaging 
for students, tied to academic standards, appropriate for 
students from diverse economic and social backgrounds, 
and fairly easy for teachers to use with a small amount of 
preparation time. 

Harcourt School Publishers adapted and expanded its 
existing school-day materials to develop Harcourt Mathlet- 
ics, a structured math model with daily 45-minute sessions 
in which students progress through material at their own 
rate, with pretests at the beginning of each topic to guide 
lesson planning and posttests to assess mastery or the 
need for supplemental instruction. The model also includes 
games to build math fluency, hands-on activities, projects, 
and computer activities for guided instruction, practice, or 
enrichment. 
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The study 

The math program was implemented in 25 after-school 
centers, chosen for their expressed interest and their ability 
to implement the program and research design. The study 
sample’s 1,961 students, in grades 2 through 5, were identi- 
fied by local staff as in need of supplemental academic sup- 
port to meet local academic standards and were enrolled in 
the after-school programs. Students were assigned by lottery 
to either the Harcourt Mathletics program or to the regular 
after-school programming. The regular programming con- 
sisted primarily of help with homework or locally assembled 
materials that do not follow a structured curriculum. 

The evaluation examines four primary questions: 

• Does the Harcourt after-school math program improve 
math proficiency over what students would achieve 

in regular after-school programs, as measured by test 
scores? 

• What are the impacts of the after-school math instruc- 
tion for subgroups of students based on their prior 
academic performance and grade? 

• Does the after-school math instruction affect other 
in-school academic behavior outcomes, as measured 
by reports from regular-school-day teachers of student 
engagement, behavior, and homework completion? 

• What does program implementation look like, and was 
it implemented as intended? 

The second and fourth questions address information to 
better target and implement the intervention. The third 
question addresses whether extended learning leads to ad- 
ditional positive or negative student academic behaviors. 

Impact findings from the first year are based on data col- 
lected from students, regular-school-day teachers, and 
school records. The Stanford Achievement Test, Tenth 
Edition (SAT-10), abbreviated battery for math, was 
administered to students at the beginning and end of the 
school year to measure the gains in achievement. A survey 
of regular-school-day teachers measured student academic 
behavior. The study also collected information about pro- 
gram implementation and student attendance. 



The study's findings after one year 

In the first year of the study, Mathletics, the math model 
put in place in 25 after-school centers, produced the follow- 
ing interim findings. 

Student math skills and other academic behaviors 

Students in the math program did experience a statistically 
significant impact on their performance on the SAT-10 
math test. The program had no positive or negative effects 
on teacher reported student behaviors during the school 
day. 

• The average math score of the Mathletics group 
increased over the school year by 35.8 scaled points, 
compared with 33.0 scaled points for the other group, 
a statistically significant difference of 2.8 scaled points 
overall (effect size = 0.06). Impacts of 2.5 and 4.3 
scaled points for subtests of problem solving and pro- 
cedures were also statistically significant (effect sizes of 
0.05 and 0.08, respectively) (figure 1). 

• Mathletics students in grades 4 and 5 scored signifi- 
cantly higher than their counterparts (effect size = 0.09), 
but those in grades 2 and 3 did not. The math program 
did not have a different impact for students in higher 
grades and in lower grades. Mathletics students who 
previously scored at the basic level scored significantly 
higher than their counterparts (effect size = 0.07), but 
those in the below basic or proficient levels did not. 
Again, the math program did not have a different impact 
for students in these three proficiency subgroups. 

• The math program did not produce statistically signifi- 
cant impacts on any of the three academic behavior 
measures: homework completion, attentiveness in 
class, or classroom disruptiveness. 

Program implementation 

The strategies supporting the math program were imple- 
mented as intended; teacher feedback and program obser- 
vations indicated few implementation challenges. Student 
participation in the program led to an estimated increase in 
math instruction of 30 percent more hours of math instruc- 
tion (49 hours) during the school year. 
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FIGURE 1 

Student growth on math test scores 



Growth from baseline (scaled score points) 

50 ■ Math program group 

■ Regular program group 
‘Statistically significant 



Total 



Problem solving 



Impact = 4.3* 




Procedures 



• Of the math program staff, 97 percent were certified 
teachers, most often with three or more years of teach- 
ing experience. More than 90 percent reported daily 
preparation of at least 30 minutes. The average student- 
to-staff ratio of 9:1 was as intended. 

• Observational data indicated that 93 percent of the 
classes used the materials and organized the transitions 
between the parts of the daily sessions as intended. 
When asked, 50 of 51 indicated challenges to varying 
degrees with using the materials, with 16 percent indi- 
cating this as a consistent problem. Based on interviews 
of the after-school teachers, the most common issue 
with implementation was maintaining the 15-minute 
rotation schedule between program activities. 

• On average during instruction days, Mathletics stu- 
dents attended the after-school program for 73 days 



(77 percent of the instruction days offered), or 12 
more days than the other group. This translated into 
an increase in math instruction of 49 hours (57 hours 
for Mathletics students compared with 8 hours for 
other students, who received regular after-school 
programming). 

Upcoming report 

The study was expanded to include a second year of imple- 
mentation and data collection in 15 of the original partici- 
pating math centers. This sample includes students who 
were part of the study in the first year and students who 
were new to the study in the second year, allowing the new 
wave of data collection to shed light both on the cumulative 
impact of the enhanced after-school program on return- 
ing students and on the impact of a more mature program 
on new students. The results will be presented in the final 
report of the evaluation. 

Note 

1. http://ies.ed.gov/ncee/pubs/20084021/exec_summary.asp 

For the full report, please visit: 

http://ies.ed.gov/ncee/pubs/20084021/index.asp 

Black, A.R., Doolittle, F., Zhu P„ Unterman, R., and Gross- 
man, J. B. (2008). The Evaluation of Enhanced Academic 
Instruction in After-School Programs: Findings After the 
First Year of Implementation (NCEE 2008-4021). Wash- 
ington, DC: National Center for Education Evaluation and 
Regional Assistance, Institute of Education Sciences, U.S. 
Department of Education. 



NCEE developed the Evaluation Briefs to offer short 
synopses of complex technical evaluation reports. This 
brief was not prepared by the study authors. 
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